Before this patch we woke up the GC service instead of recovery service
in `bucket_recovery_pause`. It could lead to a longer tests' execution
time. Now, we fix it by changing `garbage_collector_wakeup` to
`recovery_wakeup`.

NO_TEST=refactoring
NO_DOC=refactoring

Before this patch the recovery service used functions, which determine
if the bucket can be recovered, one by one in `recovery_step_by_type`.
Moreover, the function was called from `recovery_service_f` with a lot
of code duplication. In the following patches the logic of recovery
service will become more complex. In order to preserve the code
readability it was decided to:

  1) Used for-loop based checking instead of one by one based checking
     of buckets' recoverability.

  2) Join the logic of saving recovered bucket ids and changing of
     `_bucket` space into one separate function - `bucket_recover`.

The functions `bucket_recover` and `bucket_status_has_destination` will
be used in the future commits.

Needed for tarantool#214
Needed for tarantool#573

NO_TEST=refactoring
NO_DOC=refactoring

Co-authored-by: Nikita Zheleztsov <n.zheleztsov@proton.me>

This commit adds new READONLY bucket status. It's allowed to have old RW
refs, which were created before making the bucket READONLY, on top of
it. However, creating new ones is prohibited. It's the state, in which
we're waiting for RW requests to end before sending bucket. Later we'll
introduce waiting on replicas too.  For details, please
check out the following RFC: tarantool#620.

The READONLY state replaces the manual rw_lock-ing of the buckets during
bucket sending, so this code can be dropped. Also, the code for
`bucket_refrw_touch` can be also dropped: Before this commit ref has
always been created, because there was no other way to set `rw_lock` on
bucket other than ref. Now, we have "persisted rw_lock", which is the
READONLY bucket state, and we don't have to create the ref forcefully
anymore. However, we still must check, that the bucket is ok, while
sending the bucket, so during preparation phase of bucket send it was
replaced with `bucket_check_state()`.

The test from bucket_ref is removed, since it tests, that rebalancer
continues to work, when the storage has READONLY bucket status, which is
not true. Rebalancer must wait for 0 SENDING, RECEIVING and READONLY
buckets in the cluster and only after that it can try rebalancing one
more time.

Part of tarantool#573

NO_DOC=internal

The check for master is duplicated in the `bucket_send()` function. It
was done explicitly at the beginning of the function and then one more
time during `bucket_check_state()` (previously during
`bucket_refrw_touch()`).

Let's drop the first check. It is already tested in the
`storage_1_1_test.test_master_exclusive_api`.

NO_DOC=internal
NO_TEST=tested

This commit adds new functions, which are meant to be used instead of
the bucket_transfer_start/end() when sending a bucket.

This is needed for tarantool#573, where rebalancer will start to check the refs
on replicas, for which proper bucket batching is needed, since it'll be
way too costly to do that before every bucket sending.

For that to work we need to mark the bucket as `transfering` and block
new RW refs (`READONLY`) far earlier, than `bucket_send` happens. The
first one is required in order to simplify the process of picking
buckets and helps not to pick the same one twice. The second one is
essential, since we don't want new refs to happen, when we picked the
bucket for sending.

Needed for tarantool#573

NO_DOC=refactoring
NO_TEST=refactoring

Before this commit the rebalancer picked the first encountered bucket
and waited for number of rw refs on it to become 0. This doesn't work
reliably, when cluster has very long RW requests, which block bucket
rebalancing for hours: the instance picks the same buckets and timeouts,
while waiting for no refs.

In order to fix that let's prefer buckets without refs. If there's not
enough such buckets, take the remainings from the first available buckets,
as it worked before.

Closes tarantool#351

NO_DOC=bugfix

Before this patch it was possible to break replication by long RW
requests in master switches: there was a master, which processed some RW
requests, after that the master was changed (which is pretty casual
thing, when rebalancing is in progress due to high load on storages),
then a new master started sending bucket, but replica has RW ref on it,
this breaks replication and requires manual intervention from the user
(replication becomes stopped and must be manually restarted).

In order to fix that we require all replicas to be running, when
rebalancing happens. Before making a bucket SENDING it becomes READONLY
and a master waits for RW requests on replicas to end. In READONLY state
it's prohibited to create new RW requests for that bucket. For details
see the tarantool#620 RFC.

The commit is also needed for fixing the doubled buckets during master
switch tarantool#576. By syncing with all replicas we make sure, that all of them
has the bucket at least in READONLY state. So, after master switch
happens the bucket router won't be able to send RW requests to it.
Together with service, which synchronizes the `_bucket` space before any
recovery/gc/rebalancing happens, it will solve the tarantool#576.

Note, that from now on in some tests we should use `chunk_timeout`
intsead of `timeout`, since otherwise we can timeout on waiting for
synchronization and not sending a data.

Additionally, this commit introduces `sync_timeout` for `bucket_send()`.

Closes tarantool#573
Part of tarantool#576

NO_DOC=bugfix

The test is pretty flaky and fails with error "Duplicate key exists in
unique index 'pk' in space '_bucket'", when trying to force create
buckets.

Let's properly wait for all buckets to be garbage collected and not just
one, as it was before.

NO_DOC=test

Sometimes the test failed with `Not found` error, when trying to send
bucket from `replica_2_a` right after its start. The problem there is
that firstly the node should get that bucket from `replica_2_b` via
replication.

It'll be fixed automatically with introduction of the service, which
wait for `_bucket` synchronization, but we're not there yet, so let's
fix it manually.

Closes tarantool#528

NO_DOC=test

Sometimes the test case `test_basic_storage_service_info` failed with
`routes_applier_service is still nil` error. It sends a bucket and then
waits for rebalancer to send that bucket back and sometimes rebalancer
cannot start and waits, since recovery is not finished yet.

Let's wait for recovery and gc to work before enabling the rebalancer to
fasten up the process of creation apply routes worker, which are checked
in that test case.

Closes tarantool#526

NO_DOC=test